Evaluating AdApt, a multi-modal conversational, dialogue system, using PARADISE
نویسنده
چکیده
This master’s thesis presents experiences from an evaluation of AdApt, a multi modal, conversational dialogue system, using PARADISE, PARAdigm for Dialogue System Evaluation, a general framework for evaluation. The purpose of this master’s thesis was to assess PARADISE as an evaluation tool for such a system. An experimental study with 26 subjects was performed. The subjects were asked to interact with one of three different system versions of AdApt. Data was collected through questionnaires, hand tagging of the dialogues and automatic logging of the interaction. Analysis of the results suggests that further research is needed to develop a general framework for evaluation which is easy to apply and can be used for varying kinds of spokendialogue systems. The data collected in this study can be used as starting point for further research. NyckelordKeywordPARADISE, AdApt, system evaluation, dialogue system, multi-modal
منابع مشابه
Automatic Evaluation: Using a DATE Dialogue Act Tagger for User Satisfaction and Task Completion Prediction
The objective of the DARPA Communicator project is to support rapid, cost-effective development of multi-modal speech-enabled dialogue systems with advanced conversational capabilities. During the course of the Communicator program, we have been involved in developing methods for measuring progress towards the program goals and assessing advances in the component technologies required to achiev...
متن کاملEvaluation for Darpa Communicator Spoken Dialogue Systems
The overall objective of the DARPA COMMUNICATOR project is to support rapid, cost-effective development of multi-modal speechenabled dialogue systems with advanced conversational capabilities, such as plan optimization, explanation and negotiation. In order to make this a reality, we need to find methods for evaluating the contribution of various techniques to the users’ willingness and ability...
متن کاملEvaluating Spoken Language Systems
Spoken language systems (SLSs) for accessing information sources or services through the telephone network and the Internet are currently being trialed and deployed for a variety of tasks. Evaluating the usability of different interface designs requires a method for comparing performance of different versions of the SLS. Recently, Walker et al (1997) proposed PARADISE (PARAdigm for DIalogue Sys...
متن کاملObserving, Coaching and Reflecting: A Multi-modal Natural Language-based Dialogue System in a Learning Context
The Metalogue project aims to develop a multi-modal, multi-party dialogue system with metacognitive abilities that will advance our understanding of natural conversational human-machine interaction and dialogue interfaces. This paper introduces the vision for the system and discusses its application in the context of debate skills training where it has the potential to provide learners with a r...
متن کاملPARADISE: A Framework for Evaluating Spoken Dialogue Agents
This paper presents PARADISE (PARAdigm for Dialogue System Evaluation), a general framework for evaluating spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviors, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to perf...
متن کامل